This invention relates to decoding apparatuses for seamlessly playing back a video stream comprised of streams having different frame rates and, more specifically, to an apparatus for decoding a digital stream which is a mixture of interlace and progressive streams.
Satellite digital broadcasting first started in the United State in 1994, then became widespread in Japan and across Europe in 1996, and is now in full-fledged into practical use. Unlike conventional analog broadcasting, digital broadcasting can provide various services, such as high-definition, multi-channel data broadcasting. To achieve such services at broadcasters, great innovations have been introduced to broadcast station systems and receivers. Such new services under consideration include high-definition broadcasting, which is a service inherent in broadcasters, and data broadcasting, which is a new service. By the latter half of 2000, BS digital broadcasting is scheduled to be put in use in Japan.
As one approach toward high-definition, adoption of a progressive scheme prescribed in number 7 of Departmental ordinance of Ministry of Posts and Telecommunications has been considered in national CS (satellite) digital broadcasting. In the progressive scheme, the amount of information is doubled compared with the conventional interlace scheme, and extremely high vertical resolution can be achieved. Moreover, in the progressive scheme, deterioration in image quality associated with interlace operation such as flicker is prevented, and video of high image quality can be achieved.
Most of the existing video contents are created in the interlace scheme. Not all of upcoming video contents will be created in the progressive scheme, but some of them coming early would be created in the interlace scheme, and then gradually many of them will be created in the progressive scheme. Moreover, once created in the interlace scheme, the video content would never be recreated in the progressive scheme unless there is any specific reason.
With such situation, at least for the time being, the broadcasting goes on mostly with the current interlace-scheme, scarcely with progressive-scheme. Such broadcasting is called mixed-scheme-composition, in which a stream in the interlace scheme (hereinafter, xe2x80x9cinterlace stream ISxe2x80x9d) and a stream in the progressive stream (hereinafter, xe2x80x9cprogressive stream PSxe2x80x9d) are mixed in the same program. The ratio of the mixed-scheme-composition will be decreased in the future, but will not disappear for a long while.
This mixed-scheme-composition not only applies to digital broadcasting, but also to equipment that handles MPEG streams such as DVD. That is, a situation is expected to occur where a mixed digital stream comprised of contents created in the interlace scheme and the progressive scheme is recorded on a single DVD and provided. When this DVD is played back, the same situation occurs as in the above-stated digital broadcasting of mixed-scheme-composition.
When the above digital stream broadcasting of mixed programming is viewed or the DVD with the mixed digital stream recorded thereon is played back, the decoding side should correctly support MPEG streams of different frame rates, such as those in the interlace scheme and in the progressive scheme. Otherwise, played-back images will be disrupted at the time of switching of the interlace scheme/progressive scheme, as will be described below with reference to FIGS. 27 and 28. In other words, played-back images are not displayed by a unit of frame or field, or displayed incompletely. This causes visually-perceivable discontinuity of screen progress, and makes a viewer feel uncomfortable.
The operation carried out in a conventional video apparatus for decoding the mixed stream changed from the interlace stream IS to the progressive stream PS is shown in FIG. 27. In the drawing, in image display periods (hereinafter abbreviated as xe2x80x9cperiodxe2x80x9d as required for convenience) T0 to T4, an interlace stream IS comprised of a P picture P9 (period T0), a B picture B7 (period T2), and a B picture B8 (period T4) is provided through a bit-stream FIFO to the video decoding apparatus.
Then, in image display periods T6 to T11, a progressive stream PS comprised of an I picture I0 (period T6), a P picture P3 (period T7), a B picture B1 (period T8), a B picture B2 (period T9), a P picture P6 (period T10), and a B picture B4 (period T11) is provided through the bit-stream FIFO to the video decoding apparatus. The operation of the video decoding apparatus in each period is described below.
Note that a suffix xe2x80x9cPxe2x80x9d added to each picture represents Predictive Picture, xe2x80x9cIxe2x80x9d represents Intra Picture, and xe2x80x9cBxe2x80x9d represents Bidirectionally Predictive Picture. A numeral that follows each of the suffixes P, I, and B indicates a display order of that picture.
Each picture data of the interlace stream IS included in the input stream is written in a specified frame memory FMn at the same time when decoding starts in synchronization with a corresponding decode start signal. The decoded picture data written in the frame memory FMn is read therefrom in predetermined timing. In synchronization with a vertical synchronization signal (while a vertical signal is at high level), a bottom field or top field is displayed based on a display parity.
Note that, the P picture and I picture are written in the frame memory FMn, and then read therefrom for display after passing a predetermined time, while the B picture is written in the frame memory FMn and read therefrom at the same time.
Also, similarly to the picture data of the interlace stream IS, picture data of the progressive stream PS included in the input stream is decoded, written in the frame memory FMn, and read therefrom for display. However, what is displayed is not a field but a frame, irrespectively of the display parity. The operation from decoding to displaying the input stream in each image display period is specifically described below.
In the period T0, decoded data of a P picture P6 and a B picture (not shown) included in the interlace stream IS before the period T0 are written in a frame memory FM2 and a frame memory FM3, respectively. The P picture P9 is sequentially decoded while being inputted, and written in a frame memory FM1.
Then, in synchronization with the vertical synchronization signal for display, and further based on the display parity, the decoded data of the P picture P6 is read from the frame memory FM2, and the bottom field image display starts based on the display parity. Note that, in a conventional example shown herein, the interlace stream is displayed with its bottom field first.
In the period T1, after the bottom field image display of the P picture P6 started in the period T0 ends, the top field image display of the P picture P6 starts based on the display parity.
In the period T2, the B picture B7 is sequentially decoded while being inputted, written in the frame memory FM3, and then further read and the bottom field image display starts based on the display parity.
In the period T3, after the bottom field image display of the B picture B7 started in the period T2 ends, the top field image display of the B picture B7 starts based on the display parity.
In the period T4, the B. picture B8 is sequentially decoded while being inputted, written in the frame memory FM3, and then further read and the bottom field image display starts based on the display parity.
In the period T5, after the bottom field image display of the B picture B8 started in the period T4 ends, the top field image display of the B picture B8 starts based on the display parity.
In the period T6, the I picture I0 of the progressive stream PS is sequentially decoded while being inputted, and written in the frame memory FM2. Simultaneously, the decoded data of the P picture P9 of the interlace stream IS is read from the frame memory FM1, and the bottom field image display starts based on the display parity.
In the period T7, a P picture P3 of the progressive stream PS is sequentially decoded while being inputted, and written in the frame memory FM1. Note that, from the frame memory FM2, the I picture I0 of the progressive stream PS is read, and a frame image thereof is displayed. Therefore, the top field image of the P picture P9 read from the frame memory FM1 in the period T6 is not displayed. As such, instead of the top field image of the P picture P9, the frame image display of the I picture I0 of the progressive stream PS starts, which causes image distortion (discontinuity) that makes the viewer feel uncomfortable.
In the period T8, a B picture B1 of the progressive stream PS is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B1 is read, and the frame image thereof is displayed. In this way, the image of the P picture P9, which is the last display data of the interlace stream IS, is not completely displayed, and the image display is switched to the progressive stream PS.
In the period T9, a B picture B2 is sequentially decoded while being inputted, and written in the frame memory FM2. Simultaneously, the B picture B2 is read, and the frame image thereof is displayed.
In the period T10, a P picture P6 is sequentially decoded while being inputted, and written in the frame memory FM2. Further, the decoded data of the P picture P3 is read from the frame memory FM1, and the frame image thereof is displayed.
In the period T11, a B picture B4 is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B4 is read, and the frame image thereof is displayed.
As such, when broadcasting is switched from the interlace stream IS to the progressive stream PS in the image display period T5, the next image of the progressive stream PS is displayed in the image display periods T6 to T7 while the top field image of the P picture P9 of the interlace stream IS is left not being displayed.
Next, the operation carried out in the conventional video decoding apparatus (not shown) for decoding the mixed stream changed from the progressive stream PS to the interlace stream IS is shown in FIG. 28. In the drawing, in image display periods T6 to T11, the progressive stream PS comprised of a P picture P6 (period T6), a B picture B4 (period T7), a B picture B5 (period T8), a P picture P9 (period T9), a B picture B7 (period T10), and a B picture B8 (period T11) is provided through the bit-stream FIFO to the video decoding apparatus.
In image display periods T0 to T4, an interlace stream IS comprised of an I picture I0 (period T0), a P picture P3 (period T2), and a B picture B1 (period T4) is provided through the bit-stream FIFO to the video decoding apparatus. The operation of the video decoding apparatus in each period is described below.
In the period T6, the P picture P6 of the progressive stream PS is sequentially decoded while being inputted, and written in the frame memory FM2. Simultaneously, the decoded data of the preceding P picture P3 of the progressive stream PS is read from the frame memory FM1, and the image display thereof starts.
In the period T7, the B picture B4 is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B4 is read, and the frame image thereof is displayed.
In the period T8, the B picture B5 is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B5 is read, and the frame image thereof is displayed.
In the period T9, the P picture P9 is sequentially decoded while being inputted, and written in the frame memory FM1. Furthermore, the decoded data of the P picture P6 is simultaneously read from the frame memory FM2, and the frame image of the P picture P6 is displayed.
In the period T10, the B picture B7 is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B7 is read, and the frame image thereof is displayed.
In the period T11, the B picture B8 is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B8 is read, and the frame image thereof is displayed.
In the period T0, the I picture I0 of the interlace stream IS is written in the frame memory FM2 while being inputted. Furthermore, the decoded data of the P picture P9 of the progressive stream PS is simultaneously read, and the frame image thereof is displayed.
In the period T1, the frame image of the P picture P9 of the progressive stream PS displayed in the period T0 is still displayed.
In the period T2, the P picture P3 of the interlace stream IS is sequentially decoded while being inputted, and written in the frame memory FM1. Furthermore, the I picture I0 of the interlace stream IS is simultaneously read from the frame memory FM2, and the bottom field image thereof is displayed based on the display parity.
In the period T3, after display of the bottom field image of the I picture I0 started in the period T2 ends, the top field image of the I picture I0 is displayed based on the display parity.
In the period T4, the B picture B1 of the interlace stream IS is sequentially decoded while being inputted, and written in the frame memory FM3. Simultaneously, the B picture B1 is read, and the bottom field image thereof is displayed based on the display parity.
In the period T5, after the bottom field image display of the B picture B1 started in the period T4 ends, the top field image of the B picture B1 is displayed based on the display parity.
As stated above, when broadcasting is changed from the progressive stream PS to the interlace stream IS between the image display periods T11 and T0, bottom_field is displayed in the image display periods T2 to T3 even if top_field of the interlace stream IS has to be first displayed. Displaying (Outputting from the frame memory FMn) top_field and bottom_field in reversed order causes a shift in time relation between field-basis images, and makes the viewer feel awkward to watch the images.
This is because, if an image whose top_field is supposed to be displayed first with respect to a time axis is displayed with its bottom_field first displayed, the image is displayed against the time axis. Therefore, when the video stream STv is changed from the progressive stream PS to the interlace stream IS, it is of the utmost importance to determine which of top_field and bottom_field is first displayed and to make the displayed field match the display parity.
Moreover, when the progressive stream PS is decoded in decode timing for the interlace stream IS, the amount of data becomes doubled compared with a case where the interlace stream IS is processed, and therefore overflow occurs. On the other hand, when the interlace-scheme stream is decoded in decode timing for the progressive stream PS, conversely, the amount of data becomes half compared with a case where the progressive stream PS is processed, and therefore underflow occurs. As a result, decoding fails is failed.
In the examples shown in FIGS. 27 and 28, a group of frame memories is structured by three frame memories, that is, the frame memories FM1 and FM2 for storing an I picture or P picture, and the frame memory FM3 for storing only a B picture, wherein each picture is stream image data. In the group of frame memories as structured above, storing, that is, writing, image data in each frame memory FMn (n=1, 2, 3) and reading the written image data from the frame memory FMn for display are synchronized to each other in timing.
In other words, while an I picture or P picture to be stored in the frame memory FM1 or FM2 (for example, to be stored in the frame memory FM1) is decoded, a picture that has been already decoded (stored in the frame memory FM2) is read for display. When a B picture stored in the frame memory FM3 is decoded, the B picture is read for display while being decoded.
For example, in the image display period T0 of FIG. 27, the P9 picture is decoded in the frame memory FM1, and therefore the P6 picture stored in the frame memory FM2 is displayed. In the image display period T2, the B7 picture is decoded in the frame memory FM3, and therefore the B7 picture is displayed. This is because the B picture is decoded with bidirectional prediction between frames and has references from the preceding and succeeding I picture and P picture, and therefore those reference pictures have to be held in the frame memories FM1 and FM2.
Moreover, in the input streams as shown in FIGS. 27 and 28, the B picture, which is decoded and displayed simultaneously, has to be controlled in decode timing with display progress speed. In other words, in some cases, the B picture is too large to be decoded within a cycle of the vertical synchronization signal (16.7 ms), thereby causing display disruption.
In view of the above, an object of the present invention is to provide a video decoding apparatus that analyzes each frame rate and a display parity (a parameter for determining which of top/bottom_field is displayed first) in a stream comprised of plurality of video data of different frame rates and controls timing of starting video decoding, thereby achieving seamless playback without screen display disruption.
The present invention has the following features to achieve the above object.
A first aspect of the present invention is directed to a decoding apparatus for decoding a video stream comprised of a plurality of streams of different frame rates in predetermined decode timing for seamless playback. The decoding apparatus in accordance with the first aspect of the present invention comprises a frame rate extracting unit for extracting each of the frame rates before decoding the plurality of streams, and a decode control unit for determining timing for decoding a slice layer of the video stream based on the extracted frame rate, wherein the seamless playback is possible by decoding the video stream in the determined decode timing even when the video stream is switched among streams of different frame rates.
As described above, in the first aspect, decoding timing is determined based on the frame rate of the stream extracted in advance before being decoded. Therefore, an appropriate process can be ready for the stream to be decoded.
According to a second aspect, further to the first aspect, the frame rate extracting unit analyzes the video stream based on the decode timing, and separates the video stream into header information and compressed data.
According to a third aspect, further to the second aspect, the decode control unit delays, based on the header information, a vertical synchronization signal defined by a display format of the video stream for a predetermined time according to each of the frame rates of the plurality of streams included in the video stream, and generates a decode timing signal for defining the decode timing.
According to a fourth aspect, further to the third aspect, the predetermined time is shorter than one cycle of the vertical synchronization signal.
According to a fifth aspect, further to the fourth aspect, the predetermined time is a half of one cycle of the vertical synchronization signal.
According to a sixth aspect, further to the first aspect, the decode unit decodes a slice layer of compressed data and then decodes a picture layer of a following picture.
As described above, in the sixth aspect, the picture layer of the picture to be decoded next is decoded. Thus, information for displaying that picture can be obtained, and the picture can be quickly displayed.
According to a seventh aspect, further to the third aspect, the decode control unit includes, for the video stream, a decode timing corrector for alternately masking the decode timing signal for an interlace stream, and outputting the decode timing signal as it is for a progressive stream.
According to an eighth aspect, further to the seventh aspect, the decode control unit masks the decode timing signal for completing display of the interlace stream when the video stream is changed from the interlace stream to the progressive stream.
According to a ninth aspect, further to the seventh aspect, the decode control unit prohibits masking of the decode timing signal, and decodes a first picture of the interlace stream that follows a last picture of the progressive stream when the video stream is changed from the progressive stream to the interlace stream.
According to a tenth aspect, further to the seventh aspect, the decode control unit sets a multi-level transition parameter indicating a transition of the video stream to a first predetermined value when the video stream (STv) is not changed while decoded, sets the transition parameter to a second predetermined value when the video stream is changed from the interlace stream to the progressive stream, that is, when frame rate code is changed from a value less than 7 to 7, sets the transition parameter to a third predetermined value when the video stream is changed from the progressive stream to the interlace stream, that is, frame rate code is changed from 7 to a value less than 7, and sets the transition parameter to the first predetermined value in next decode timing.
According to an eleventh aspect, further to the tenth aspect, the decode control unit masks top_field of the decode timing signal when the video stream is the interlace stream and video display is carried out sequentially from top_field in accordance with video display parity, and masks bottom field of the decode timing signal when video display is carried out sequentially from bottom field in accordance with the video display parity.
According to a twelfth aspect, further to the eleventh aspect, whether to mask top_field or bottom_field of the decode timing signal is defined by a binary mask signal parity mask.
According to a thirteenth aspect, further to the twelfth aspect, the decode control unit reverses parity_mask for parity correction when parity_mask is not equal to the video display parity and frame_rate_code is smaller than 7, and top_field_first is equal to the video display parity.
According to a fourteenth aspect, further to the seventh aspect, the decode control unit sets a binary initial parameter defining an initial state of the decode control unit to a first value.
According to a fifteenth aspect, further to the fourteenth aspect, when decoding starts, the decode control unit always decodes a first picture of a following stream in order of the picture layer, the slice layer, and then the picture layer when the initial state parameter indicates the first value.
According to a sixteenth aspect, further to the seventh aspect, when decoding starts, the decode control unit always executes a sleep process after setting the video stream to the progressive stream.
According to a seventeenth aspect, further to the seventh aspect, the decode control unit carries out frame rate correction at a time of a first transition of the video stream from a 24-frame interlace stream to either of a progressive stream or 30-frame interlace stream and at a time of a second transition of the video stream in reverse of the first transition. and sets a binary parameter defining parity of the stream from one value to another value for parity correction at a time of 3:2 pull-down.
According to an eighteenth aspect, further to the fifteenth aspect, the picture layer includes sequence_header, GOP_header, and Picture header, the slice layer includes a bit stream having slice_header and thereafter, and the picture layer and the slice layer forms one image.
According to a nineteenth aspect, further to the seventh aspect, the decode control unit ends decoding when detecting sequence_end at a tune of decoding the picture layer, and when detecting sequence end of the video stream (STv), decodes the picture layer, the slice layer, and then the picture layer, and then executes a sleep process.
According to a twentieth aspect, further to the fourteenth aspect, the decode control unit prohibits output image setting when the initial state parameter indicates the first value.
According to a twenty-first aspect, further to the seventh aspect, the decode control unit makes a previously-decoded I picture or P picture displayed at a time of underflow at which decoding in previous decode timing has been incomplete when another decode timing comes.
According to a twenty-second aspect, further to the seventeenth aspect, when underflow occurs at a B picture, the decode control unit decodes, a picture layer and a slice layer of a following I picture or P picture, and then a following picture layer, and then executes a sleep process.